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Recommendations to increase the relevance of 
information, improve the communication of information, and increase 
user responsiveness to information are presented. Recommendations 
include the following: (1) the Department of Education (ED) should 
institute a flexible planning system for evaluations of federal 
education programs; (2) congressional requests for evaluation should 
identify the kind of question (s) to be addressed; (3) in evaluation 
initiated by the ED, the kinds of evaluation activities to be carried 
out should be specified clearly and should be justified in terms of 
program development or program implementation; (4) the ED should 
ensure that dissemination of evaluation results achieves adequate 
coverage; (5) the ED should observe the rights of any parties at 
interest and the public in general to information generated about 
public programs; (6) the ED should test various mechanisms for 
providing linkage between evaluators and potential users; and (7) the 
ED should provide funds for training programs in evaluation to 
increase the skills of individuals currently charged with carrying 
out or using evaluations and to increase the participation of 
minorities. While no formal recommendation was made to increase user 
responsiveness to evaluation, it was suggested that the ED consider a 
plan requiring evaluation use reports. (RL) 
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A frequently voiced criticism is that evaluation findings are seldom 
used. Implicit in this criticism is the notion that utilization means direct 
and often immediate changes in policy and program. In fact, there are 
several different types of utilization, not all immediately apparent. More- 
over, the systematic communication of findings does not automatically lead to 
utilization, nor does utilization always imply change. 

Evaluation findings may, indeed, be used for making specif indecisions at 
a given time, but they may also enter into decisions where other factors lobby 
for greater influence. Evaluation sponsors may not have intended to use the 
evaluation report for specific and immediate changes. Even when no immediate 
use of knowledge clearly derives from evaluations, the effect of that knowledge 
cumulates over time and is slowly absorbed, eventually leading to changes in 
concepts and decision perspectives. On the other hand, evaluation information 
q may be considered and discarded because it is inappropriate or its suggestions 

for policy infeasible. 

The. Committee on Progr Evaluation in Education has considered how to 
jt increase evaluation use in federal education programs. Its recommendations 

are easily categorized into three groupings: (1) increasing the relevance of 



♦Paper presented as a part of a symposium titled "Program Evaluation: What? 

How? To What Ends?" American Educational Research Association, Los Angeles, 
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information; (2) improving the communication of information; and, (3) increas- 
ing user responsiveness to information. These categories are not unique to 
the work of the Committee, but rather draw heavily upon the research 
literature on knowledge and evaluation utilization (see, for example, 
Davis and Salasin 1975, Glaser 1973, Havel ock 1976, Caplan 1977, Patton 1978, 
Al kin et al . 1979). For each category, the recommendations from the 
utilization chapter of the report will be presented, followed by applicable 
recommendations from other report sections. 

INCREASING THE RELEVANCE OF INFORMATION 
For evaluation information to be used it must be perceived as "relevant" 
by the primary user audience. Achieving such a state of perceived relevance, 
however, is not easy: relevance demands continuous interaction between tne 
primary audience and the researcher. Evaluator and audience must ne gotiate 
content , deciding together what the evaluation can and should address and how 
to report data. The evaluator cannot assume that content won't need 
renegotiation several times. When the evaluator understands the audience's 
aims, they must devise an appropriate research form together. The evaluation 
report must offer realistic options which pay attention to various constraints 
on the user audience. Finally, the evaluation information must be timely , 
addressing current issues and observing immediate schedules. 

Utilization Chapter Recommendations 

Recommendation D-10. The Department of Educ ation should institute a 
fl exible planning system" for evaluations of federal education programs . 

A flexible planning system maximizes the likelihood that evaluation 

information will yield relevant information in a timely manner. The 

Committee believes that an evaluation plan for a ;ia.ior education program 



should contain a series of linked studies: some which furnish factual 
information in reasonably short time, and some which address issues requiring 
longer study. Thus, information will be available for recurring legislative 
decision cycles on education programs, while other ongoing evaluation 
studies address long-range problems, (e.g., interesting but unanticipated 
questions that arise as a result of ongoing research, changes in policy, 
or development of new programs). 

Other Recommendations 

Recommendation C-l. Congressional requests for evaluation should 
identify the kind of question(s) to be addressed" ~ 

The Committee believes that evaluators can provide more relevant 

information if the Congress makes more explicit the nature of its information 

needs (e.g., prcgram services, program coverage, program impact). When 

Congress does not specify questions and audiences ahead of time, the evaluators 1 

systematic engagement in continuing dialogue with members of Congress and 

congressional staff can clarify congressional intent. 

Recommendation D-l. In evaluations initiated by the Department 
of Education, the kinds of evaluation activities to be carried 
ouFshould be specified clearly and snould be justified in terms 
of program development or program implementation . — ~~ 

This recommendation, analogous to recommendation C-l made to the 

Congress, emphasizes specifying the evaluation activities appropriate to a 

program's given stage of planning or implementation. Department officials 

need to establish what they wish to know about a program, within what time 

periods, and the potential for the evaluation information. 

IMPROVING COMMUNICATION OF INFORMATION 
Before evaluation can be used it must be communicated. And the 
conscious, systematic efforts of evaluators can improve communication. 



Therefore, evaluators must establish clearly that attention to deliberate 
communication is not a pro forma exercise. The literature identifies many 
factors for enhanced communication and evaluation use which can be grouped 
under three headings: coverage , communicability , and linkage . 

Communication coverage is attained if relevant evaluation information is 
delivered to primary audiences, and available and distributed to appropriate 
secondary audiences. 

Communicability is best attained by matching the evaluators communication 
style to that of the primary audiences(s) . Several factors contributing to 
communicablity have emerged from the literature and successful practice: 
(1) reports which are understandableand situationally applicable; (2) emphasis 
on the positive action steps; (3) personal presentation by a credible evaluator. 

Prcvidmg the necessary linkage between evaluation findings and action 
is a problem of great concern. Research on linkage has focused on a number 
of factors: (1) responsiveness to differences between researchers and 
audience; (2) mediating problem definitions between researchability and 
audience relevance; (3) human agents who facilitate or perform the requisite 
face-to-face communication; (4) open systems which encourage people to be 
more aware and accepting of evaluation information that may come from the 
outside J Evaluators can perform linkar*. Minctions themselves, but occasion- 
ally need external "transfer agents." Ihe recommendation as stated is 
attentive to the primary orientation of most. people who evaluate national 
programs; They are, at best, able - but not user oriented. - researchers. 
Linkage agents is one solution to the current deficiencies; a second is to 
train user oriented evaluators capable of serving both the traditional 

lOn a personal note, I add that I see the knowledge transfer agent as 
redundant if evaluation is properly done and evaluators are themselves 
more user oriented (see Alkin et al, 1979; Patton, 1978). Indeed, the 
inclusion of this discussion predefines evaluators as pure researchers 
oblivious to policy settings (but knowledgeable. as statisticians, research 
designers, etc. ) 
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evaluation research function and the user linkages. 



Utilization Chapter Recommendations 

Recommendation D-13. fh& Department of Education should ensure 
tha t dissemination of evaluation results achieves adequate 
coverage . 

At the very least, evaluation results must be communicated (delivered) 

to the primary audiences(s). This requirement would seem self-evident, but 

it often is not met. Beyond that, evaluation results should be reported to 

a variety of secondary audiences. In the case bfwitten final reports this 

would certainly require the avail ablity and distribution to these interested 

groups. To insure attention to attaining adequate coverage, the Committee 

believes that all evaluation §tH^e*8 ^should include a dissemination plan 

oriented towards maximizing the likelihood of use. This plan should 

include: primary and secondary audience specification, information needs of 

each, dissemination strategies to be employed, timetables, etc. 

Recommendation D-14. The Department of Education should observe the 
rights of any parties at interest and the public in general to 
information gener a ted about public programs . 

Though minimal dissemination is concerned primarily with the immediate 
or primary audience, other people are significantly affected by the programs 
being evaluated; for example, those who manage them, those who provide k v 
program services, and those who are intended to benefit (or their represen- 
tatives). Since evaluations are paid for with public funds, they should 
also be made available to the public at large. Steps needed to provide 
improved public access to evaluation findings include: (1) limiting the 



2 A further note is in order: attention to external linkage agents focuses 
on what to do to facilitate the use of a completed evaluation; the primary 
linkages are established prior to the evaluation study's conduct and are 
best made by the evaluator. .(A primary issue here is evaluator credibility.) 



period during which the sponsoring agency or managers and executives to 

exclusive access to evaluation findings; (2) maintaining the privacy rights 

of individuals and organizations; and (3) . appending interpretations and 

critiques issuing from* a review to evaluations on public release. 

Recommendation D-9. The Department of Education should te st 
various mechanisms for jarcviding linkage between evaluators 
and potential usefl T 

The Committee recommended that the Department consider establishing 
a unit charged witiTstudying, developing and instituting knowledge transfer 
mechanisms and evaluating tneir effectiveness. Alternatively, outside v 
experts might be charged with this responsibility. Appropriate activities 
would include: Translating evaluation reports so that they can be under- 
stood by the intended audiences (see footnote #1 on page 4); funding 
research (in conjunction with the NIE dissemination research unit) on the 
communication and use of evaluation information , and developing procedures 
designed to improve the day-to-day use of evaluation data, at least within 
the department. 

Other Recommendations 

Recommendation D-4. The Department of Education, should provide 
f u n d s for training programs in evaluation to increase the skills 
oflndividuals current l y charged with.carrying out or using 
evaluations and to increase the participation of minorities . 

The communications gap between evaluators and administrators can be 

narrowed substantially without the aid of external linkage agents by 

appropriate training for evaluators. Evaluators need exposure to the 

problems, procedures, and constraints of federal education programs. They 

also need to improve interpersonal and communications skills in order to 

convey evaluation information effectively. Finally, evaluators need to be 



convinced during their training of the importance of having an orientation 
to user needs. 

INCREASING USER RESPONSIVENESS TO EVALUATION 
Recently Sechrest (1980) has suggested that, if high-level administrators 
could be trained in how evaluations are done and how researchers present 
results, utilization would be increased. We have some doubts whether 
training alone is a sufficient strategy for increasing user responsiveness 
to evaluation. Nevertheless, if federal executives developed greater facility 
for the language of evaluation, they would certainly become more sophisticated 
readers of evaluation reports. Recommendation D-4 (discussed above) 
focusing on the communication gap between evaluators and users, does, in 
fact, suggest that evaluation training sessions for executives and program 
staff might sensitize them to the nature of evaluation information and 
its uses. The Committee suggests that short training sequences on such 
topics be developed and made routinely available to new staff. 

Other Comments 

User responsiveness to evaluation is further heightened by developing 
incentives for program managers to consider such information. If it is 
important to do evaluations, then it is equally important to encourage their 
use and consideration. This does not imply that recommendations in eval- 
uation reports must be followed, but that evaluating at least be "listened 
to" and weighed m the decision making process. 

While no formal recommendation was made, the Committee nevertheless 
suggested that the Department consider a plan for requiring evaluation Use 
reports . Within a year after receiving an evaluation report, federal 
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program managers would return an evaluation use report. This report would 
include, for example, how they had - or had not - used the evaluation report, 
how much, and why/why not. The manager might analyze the factors which 
impeded or encouraged use and the other data sources which influenced 
decisions. 
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RECOMMENDATIONS FOR 
IMPROVING EVALUATION UTILIZATION 
IN FEDERAL EDUCATION PROGRAMS 
(From the Report of the National Research j Council 
Committee on Program Evaluation in Education) 

T . Pnmi , ttBP nn Prnaram Evaluation in Education has considered how to 

user responsiveness to information. 

INCREASING THE RELEVANCE OF INFORMATION 

Rornmmendation D-10. The De partment of Education s hould institutes 
jgg^ ot federa l education programs. 

Recommendation C-l. fongjgs^p nal requests for evaluation should 
identify the kind ofquestion(s) to be addr essed. 

l^?Tfe7i *WW and shou'^ h. un ified i n terms of D-caCM! 
•aevelopm ent o r program i mplementation. 

IMPROVING COMMUNICATION OF INFORMATION 

Recommendation D-13. The Department of Education should ensure that 
SSnftirof eval uation results achieves adequate covgrafle. . 

Recommendation D-14. Tho nepar tment of Education shoul d observe the 
mg^'n~ qenerated about public programs. 

of individuals curW T^ 

INCREASING USER RESPONSIVENESS TO EVALUATION 
While no formal recommendation was .made, the Committee nevjrthjlljss^ 

iSBff £?: jffss ^ ^awsjsaBsw^ 

managers would return an evaluation use report. 

handout prepared by HaT^Tc^kin for P™,entation as j^^'^J-, 
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